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Abstract 


We have applied formal experiment design and 
analysis to optimize the measurement of temperature in 
a supersonic combustor at NASA Langley Research 
Center. We used the coherent anti- Stokes Raman 
spectroscopy (CARS) technique to map the temperature 
distribution in the flowfield downstream of an 1 1 60 K, 
Mach 2 freestream into which supersonic hydrogen fuel 
is injected at an angle of 30 degrees. CARS 
thermometry is inherently a single-point measurement 
technique; it was used to map the flow by translating 
the measurement volume through the flowfield. The 
method known as “Modern Design of Experiments” 
(MDOE) was used to estimate the data volume 
required, design the test matrix, perform the 
experiment, and analyze the resulting data. MDOE 
allowed us to match the volume of data acquired to the 
precision requirements of the customer. Furthermore, 
one aspect of MDOE, known as response surface 
methodology, allowed us to develop precise maps of 
the flowfield temperature, allowing interpolation 
between measurement points. An analytic function in 
two spatial variables was fit to the data from a single 
measurement plane. Fitting with a cosine series 
bivariate function allowed the mean temperature to be 
mapped with 95% confidence intervals of ±30 K, 
comfortably meeting the precision requirement of ±50 
K specified prior to performing the experiments. We 
estimate that applying MDOE to the present experiment 
saved a factor of five in data volume acquired, 
compared to experiments executed in the traditional 
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manner. Furthermore, the precision requirements could 
have been met with less than half the data acquired. 
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df 
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MSE 

PE 

PIHW 
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confounding 
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ratio of selected mean square values 
number of regressors in a model (p- 1) 
kelvin 

number of data points used to fit a 

response surface model 

number of parameters in a model, 

including intercept (£+1) 

radial axis from center of fuel jet, in 

polar coordinates 

Ratio of explained to total sum of 

squares 

spanwise axis in duct 
vertical axis in duct 
streamwise axis in duct 
standard error in the regression 
azimuthal axis about center of fuel 
jet, in polar coordinates 
analysis of variance 
coherent anti- Stokes Raman 
spectroscopy 

confidence interval half width 
degrees of freedom 
lack of fit 

modem design of experiments 
residual mean square error 
(unexplained variance) 
pure error 

prediction interval half width 
Response Surface Methodology, 
Response Surface Modeling 
sum of squares 

Executing an experiment so that the 
change in response cannot be 
uniquely attributed to a specific factor 
A threshold F statistic indicating 
minimum statistical significance 
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level 

Specific setting of an independent 
variable 

LOFF 

ratio of lack of fit to pure error 
components of unexplained variance 

model F 

ratio of model mean square to 
residual mean square 

model mean 

square 

variance that can be explained by 
model 

p statistic 

probability that a corresponding F 
statistic is not significant 

parameter space, 

design space 

Range of independent variable levels 

population 

a theoretical construct representing 
conceptually all the possible 
observations of a system 

sample 

a discrete number of observations 

site: 

Specific location in the design space 
(specific combination of independent 
variable levels) 

Introduction 


Optical measurement techniques are frequently 
used to study combustion, chemical reactions, plasmas, 
and aerospace-related flow phenomena. 1 These 
techniques can be characterized as either single-point or 
imaging techniques. Single-point techniques are 
generally more accurate and can provide simultaneous 
measurements of multiple gas properties compared to 
imaging techniques. Imaging techniques, on the other 
hand, provide spatial visualization of the flowfield, 
which is key to understanding important fluid 
mechanical and chemical processes. A common 
approach for mapping flowfields with high 
measurement precision is to translate a single-point 
technique’s measurement volume around the flowfield 
of interest, assuming flow repeatability. Unfortunately, 
this is a time-consuming process that makes this 
method of testing in wind tunnels very expensive. This 
paper investigates a method that would optimize this 
strategy, allowing a substantial reduction in cost. The 
method is known as “Modern Design of Experiments,” 
or MDOE. Specifically, we have used MDOE to 
optimize single-point coherent anti- Stokes Raman 
scattering (CARS) temperature measurements in a 
supersonic combustor. 

The foundations of MDOE go back to the early 
part of the 20 th century where formal experiment design 
was developed for agricultural experiments by Fisher 
and others. 2 Their methods allowed experiments to be 
designed and executed that minimize the effects of 
systematic errors. In the 1940s, Box and coworkers 3 
developed “response surface methods” wherein an 
analytic model was fit to the experimental data and then 
tested. 3,4 This advancement allowed substantial 


improvements in precision compared to the state of the 
art at the time. Taguchi and others popularized the 
formal experiment design methods in the late 1970s and 
1980s. Many industrial corporations adopted Taguchi’ s 
methods and some forms of his methods are still in use 
today. However, the aerospace instrumentation 
industry made very little use of formal experiment 
design and analysis until the mid-1990s. At that time, 
MDOE began to be adopted by a few researchers at 
NASA and elsewhere. Today the methods are gaining 
acceptance in the aerospace instrumentation industry. 
Since 1997, for example, MDOE has been used in over 
40 (mainly wind tunnel) tests here at NASA Langley 
Research Center. 5 MDOE is used to optimize 
parametric studies while designing scramjet engines. 6 
Lockheed Martin recently used MDOE to optimize 
conformal fuel tanks on an F-16. 7 NASA Langley 
Research Center recently began using MDOE to 
calibrate its model balance systems, replacing a method 
that had been used for several decades. 8 MDOE has 
been used in these wide ranging applications for a 
variety of reasons to be outlined in detail below. These 
have to do with reduced costs, improved accuracy, 
deeper insights into the underlying processes, and 
avoidance of systematic errors. 

In the field of laser-based measurement techniques, 
great effort is expended to improve the measurement 
precision of a technique by a factor of two. When such 
advancement is made, the community at large quickly 
adopts it. For example, modeless dye laser designs 9,10 
allowed the precision of CARS temperature 
measurements to be improved by about a factor of two. 
Now, the majority of CARS experiments being 
performed today (at least in the USA) use that 
technique. Considering the past successes of MDOE 
and the continued desire to improve measurement 
precision, we have applied MDOE to optimize the 
measurement of temperature in a supersonic combustor 
with the CARS technique. We believe that this is the 
first such application of MDOE to optimize a single- 
point laser-based measurement experiment. 

Conventional Experimental Design and Execution 

Before discussing MDOE in detail, conventional 
experimental design and execution will be reviewed 
briefly. Conventional laser-based measurements often 
begin with a loosely defined plan to “characterize a 
flowfield as well as possible” given the constraints 
(time, money, hardware limitations, etc). Measurement 
locations are often determined in an ad hoc manner, 
wherein the experimentalist attempts to “look where the 
interesting flow physics is.” Measurements are usually 
taken sequentially to minimize the time required to 
move between set points (locations in the flow). For 
example, the measurement point is sequentially stepped 
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across the flowfield of interest. Care is often taken to 
obtain enough repeated measurements to produce a 
statistically significant mean and standard deviation of 
the data allowing turbulence parameters to be 
determined. The experiment frequently ends when 
either the allotted money runs out, time runs out, or the 
experimentalist decides (often arbitrarily) that he has 
acquired enough data. 

The primary product of this research is usually a 
thesis, a conference paper, or a journal article that 
presents graphs, tables, probability distributions, and/or 
contour maps chosen to communicate the important 
physics of the flowfield. Uncertainty analysis is 
sometimes an afterthought. Other scientists wanting to 
obtain these data either need to contact the authors for a 
software copy of the data set or otherwise scan in or 
type in tabulated data from the paper. 

Modern Design of Experiments (MDOE) 

Modern design of experiments recommends a 
different approach on many of the issues described in 
the previous section. For example, a proper MDOE 
experiment design begins with a quantitative statement 
of the objective or objectives of the experiment. The 
objective is formulated after a great deal of 
consideration because the design of the experiment will 
depend strongly on this objective. For example, it is 
critical to know how the data will ultimately be used 
(e.g., compared with a computational fluid dynamics 
code). This aids in formulating specifications that can 
be expressed in the specific objectives and which help 
define quantitative exit criteria. 

Once the objectives have been determined, an 
experiment can be designed that is optimized for 
achieving these specific objectives in the least time and 
with the lowest cost. Considerations during the design 
phase of the experiment include: identification of the 
range of the independent variables to be investigated 
(the parameter space), the selection of the measurement 
sites, and the volume of data to be acquired. See, for 
example, Eq. (1), which was used in this experiment to 
estimate data volume requirements as will be described 
more fully below. § If the cost to acquire the minimum 
volume of data needed to meet test objectives exceeds 
the budget allotted for the experiment, then one knows 
prior to the start of the experiment that either the 
resources must be increased or the objectives of the 


§ Data volume specifications vary according to the specific 
requirements of the experiment. The equation referenced here 
applies to the common situation (which applied in the current 
study) where a specified precision is the dominant 
requirement. Other variations on this formula would be used 
to estimate data volume requirements in other circumstances, 
such as when the dominant requirement is to resolve specified 
differences in one or more response variables. 


experiment must be scaled back. This knowledge alone 
can save considerable waste of resources. The result of 
the design phase of the experiment is a test matrix that 
describes the combinations of independent variables 
that will be tested. 

During execution of the experiment, MDOE 
markedly differs from the conventional approach. In 
conventional experimentation, independent variable 
levels are typically set for convenience in some 
monotonically increasing order. For example, to 
measure the temperature distribution across a duct, the 
conventional approach is to start on one side of the duct 
and progress systematically across until the opposite 
side is reached. Unfortunately, this means the 
independent variable is changing systematically with 
time. It is therefore impossible to distinguish response 
changes that are due to the changes in independent 
variable from changes due to any other factor that 
might be changing with time. In MDOE testing, these 
systematic patterns are avoided. Instead, the set points 
determined in the design phase are executed in random 
order. Randomization decouples changes in 
independent variables from time. This enables us to 
distinguish the legitimate independent variable effects 
we seek to study from such time-varying systematic 
error sources as instrumentation drift, thermal effects, 
etc. 

To illustrate the benefits of randomization, we have 
numerically simulated a temperature-measurement 
experiment. Suppose it is desired to measure the 
(quadratic) temperature profile across a duct. 
Furthermore, suppose that during the time we obtain 
our measurement, the temperature everywhere in the 
duct is increasing by 20 K between measurement points 
because of an undesired, uncontrollable long-term 
temperature drift. Using the conventional approach 
would produce the results shown in Fig. 1. A quadratic 
curve fit to the measured temperatures is shifted away 



Fig. 1. Simulated temperature distribution in a duct. 
Measurements performed in sequential order while the 
temperature is increased 20 K between measurements. No 
random noise is superimposed. 
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Fig. 2. Simulated temperature distribution where the order of 
execution of the experiment has been randomized. The 
temperature is increasing 20 K between measurements. No 
random noise is superimposed. 

from the true average temperature. This is because the 
data on the left side of the graph falls below the true 
average temperature and the data on the right lies 
above. During execution of this experiment, the 
measurement location was confounded with time, 
which has induced a systematic error in the results. 

Performing the same experiment with the same 
measurement locations chosen in random order yields 
results shown in Fig. 2. The scatter in the data is larger 
than in Fig. 1 because randomization has converted the 
systematic error into an additional component of 
random error. However, a quadratic curve fit through 
the data shows that the true average temperature was 
recovered, as desired. We can always acquire more 
data to reduce the random error if the precision 
requirements of the test so dictate. The important point 
is that randomization allows us to recover the true 
relationship between the independent and dependent 
variables, which is the raison d’etre for empirical 
investigations. Note also the important fact that 
randomization defends us against all systematic errors 
variations, whether foreseen or not. 

Replication is another important tactic that is 
implemented during the execution of an MDOE 
experiment. Replication is the process of obtaining 
repeated measurements at the same set point of 
independent variables (same x, y location, for example). 
True replicates cannot be obtained one after the other. 
Instead the set point must be changed to another value 
and later returned to the original measurement point to 
provide an opportunity for all potential sources of 
random error to occur, including set point error, for 
example. 

Replication has two benefits. First, repeated 
measurements increase the data volume. The 

uncertainty in the final result decreases with the square 
root of the number of measurement points under 
commonly occurring experimental conditions. The 

second reason for replication is that not all of the 


variance in an ensemble of data can be attributed to 
known changes in the independent variables. After 
accounting for all known independent variable effects, 
there is always some residual unexplained variance that 
is responsible for uncertainty in the experimental 
results. Replication allows us to partition this 
unexplained variance into a component attributable to 
random error in the data, and another component that is 
attributable to analytical errors in defining the 
relationship between the dependent and independent 
variables. We call these two components of 
unexplained variance “pure error” and “lack of fit”, and 
we will have considerably more to say about them 
below in the discussion on response surface 
methodology. 

A third tactic that can be implemented during 
execution of an experiment is called blocking. 
Blocking involves partitioning the data set into blocks, 
usually differentiated by time. For example, during an 
experiment performed over several days, the data could 
be blocked by day, which would allow day-to-day 
variations in the experiment to be identified and 
accounted for. Without blocking, these day-to-day 
variations would appear as systematic errors. 

After the experiment is completed the data is 
analyzed using response surface methods (RSM). 3 " 4 5 A 
response surface is an analytic model having p 
adjustable constants that are fit to the n measurement 
points, often using the method of least squares. Each 
proposed mathematical model is fit to the data and then 
evaluated with a variety of different statistical tests and 
other criteria. Once a model passes these tests it is a 
candidate for testing against data obtained during the 
experiment but which were withheld from the fit. The 
model must successfully predict the test data a 
statistically significant number of times to be 
considered a valid model representation of the data. If 
a linear regression model passes all these tests, then the 
statistical uncertainty in the mean prediction of the 
model, averaged over the range of independent 
variables, can be estimated from: 

CIHW 95% - 2<j{pl nf 2 ( 1 ) 

where CIHW95% is the 95% confidence interval half 
width and a is the fit standard error, which is 
independent of the model if the model is a good fit to 
the data. In the limit that p = 1, so that only the mean 
value of the data is determined, this formula predicts 
the familiar 95% confidence interval half width for the 
sample mean. Typically, more complicated models are 
used, ranging from linear (p = 2) to quadratic (p = 3) to 
higher order polynomials, to series expansions such as 
Fourier, Chebychev, and others. Note that while this 
equation computes a confidence interval averaged over 
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Fig. 3. Partitioning of Variance. A and B are arbitrary 
independent variables. 


points used to fit the model, the actual confidence 
interval varies over the surface. The CIHW 95 o /o is 
typically smaller near the middle of the range of 
independent variables and larger near the edges where 
there is less data. 

One method used to evaluate the fit quality of 
potential models is known as analysis of variance , or 
ANOVA. 3,4,11 In general, variance provides a 
quantitative measure of the variation in a quantity. To 
quantify the variance the squares of the difference 
between each sample and some reference quantity are 
added (producing a “sum of squares”, SS) that is 
divided by the number of degrees of freedom, df. The 
df represent the minimum points needed to uniquely 
quantify the sum of squares. Variance is also called 
“mean square error.” To compute the “total variance” 
the reference quantity is the mean of all the samples. 
Alternately, one can compute the “unexplained 
variance” by using the model prediction as the 
reference quantity. Similarly, the “explained variance” 
can be determined by computing at each measurement 
location the difference between the value of the model 
and the mean value of the model averaged over all 
measurement locations and dividing by the number of 
degrees of freedom. In this way, the variance can be 
partitioned into its components, which helps quantify 
the quality of the fit. Partitioning of variance also helps 
us to interpret the model to derive a better physical 
understanding of the process being studied. Figure 3 
summarizes the partitioning of variance and shows that 
the total variance can be partitioned into explained and 
unexplained components. The explained variance can 
further be partitioned into the individual model 
parameters. This can be useful, for example, to 
determine which model parameters dominate the fit, or 
to determine which interactions between independent 
variables are significant and which are not. 

Partitioning the unexplained variance into “lack of 
fit” and “pure error” components is an important part of 
determining fit quality. The pure error component of 


the unexplained variance is determined from replicated 
measurements. Pure error variance quantifies the 
chance variation inherent in the system. An accurate 
estimate of the pure error is important because the part 
of the unexplained variance that cannot be attributed to 
pure error must be attributed to lack of fit. Lack of fit is 
the inability of the model to fit the data accurately. If 
the lack-of-fit component of the unexplained variance is 
large compared to the pure error component, the 
adequacy of the model is called into question. 

Partitioning of variance provides several figures of 
merit from which the quality of the fit can be judged. 
The Model F-statistic is defined as the ratio of 
explained variance to the unexplained variance. This 
parameter provides an estimate of signal-to-noise ratio, 
which we would like to be a large number. A common 
rule of thumb for an adequate signal-to-noise ratio 3 is 
that the F-statistic should be greater than 10* F criticah 
where F critica i is tabulated in standard statistical tables 
for various combinations of model and residual degrees 
of freedom. It represents the smallest ratio of explained 
to unexplained variance that can be resolved with a 
specified level of confidence. Figure 4 shows a 
computation of F critica i for a range of experimental 
parameters that would be expected in the present 
experiment. A confidence level of 95% has been 
assumed. F critica i varies with the number of terms in the 
model but was generally less than five for the models 
examined in the present study. So, any fit having an 
F-statistic >50 passed this test. 

Another figure of merit is the Lack-of-Fit (LOF) 
F-statistic. This is defined as the ratio of the LOF 
variance to the pure-error variance. If this ratio is near 
unity then there is unlikely to be significant lack of fit. 
In other words, statistically, it is a good fit. An 
associated statistic is called the LOF p-statistic , which 
describes the probability that the measured LOF 
F-statistic could be as large as it is due just to random 
error. A small LOF p-statistic (less than 0.05 by one 
common convention) argues against a good fit. 
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Fig. 4. Computation of F critica i for the F-statistic test. 
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Fig. 5. Simulated temperature data to illustrate the concept of 
lack of fit. Random noise with a standard deviation of 100 K 
of has been added to the same “true average temperature” 
shown in Fig. 1. 

Figures 5 and 6 illustrate the concepts of lack of fit 
and signal-to-noise ratio (. Model F -statistic test). 
Figure 5 shows two models that have been fit to a set of 
simulated experimental data that includes random 
noise. Though the first-order model represents a best 
least squares fit to the data, it shows significant 
deviation from the data at most locations. In particular, 
the deviation between the first-order model and the data 
is much larger than the scatter in the data, so the LOF 
F-statistic would be large for this fit. The second-order 
model, on the other hand, fits the data very well. More 
of the difference between the model and the data is due 
to chance variation in the data than to lack of fit, so the 
LOF F-statistic would be relatively small. 

Figure 6 shows an example where the scatter in the 
data is much larger (500 K standard deviation). In this 
case the Model F-statistic is not large enough to exceed 
our criterion of 10 *F criticia i and the model fails the lack- 
of-fit test. The large scatter would be revealed by the 
pure-error component of the unexplained variance, and 
would suggest that more data might improve the fit. If 
the pure error were small, we would have to attribute 
the lack of fit to an inadequate model, and would 
therefore have to fit a more elaborate model - typically 
one with higher-order terms. 

Another figure of merit is R 2 . This is the ratio of 
the explained sum of squares to the total sum of squares 
(where the sum of squares is equal to the variance times 
the number of degrees of freedom). R 2 provides an 
estimate of the fraction of the variance that is explained 
by the model. For example, if R 2 = 0.8, then the model 
is said to explain about 80% of the variance present in 
the data. 

Graphs of residuals also provide information about 
fit quality. Residuals are the difference between sample 
measurements and the model prediction. Residuals 
should not show any trends when graphed against any 
of the independent variables, the model predictions, or 



Fig. 6. Simulated temperature data to show failure of the 
Model F-statistic test. Random noise with a standard 
deviation of 500 K of has been added to the same “true 
average temperature” shown in Fig. 1. 

time. If substantial trends are observed in residuals 
then the model probably does not represent the data 
adequately. Large residuals (“outliers”) should be 
carefully examined, but should not be deleted without 
justification. All information provided by the data 
about the quality of the fit is carried in the residuals, 
and while an outlier may signify a “bad” data point, it 
may also identify an important deficiency in the model. 

If the model passes all the above tests, one needs to 
determine whether the uncertainty, specified for 
example by Eq. (1), meets the test requirements. If the 
requirements are not met, then it is possible that another 
model could be found that would fit the data better, 
resulting in an acceptably low level of uncertainty. 
(Note that this implies an a-priori definition of 
“acceptable,” which is a standard requirement in 
MDOE testing.) A better model can often be found by 
transforming either the dependent variable or one or 
more of the independent variables. This can improve fit 
quality or reduce the number of parameters in the 
model, thereby reducing the uncertainty. 

If the model meets the precision requirements then 
it is a candidate for testing against data withheld from 
the fitting process. Typically, ~5% of the data is 
withheld to assess model adequacy. The model must 
successfully predict the temperature in a statistically 
significant fraction of trials for the model to be 
accepted. The model is said to successfully predict the 
value of a measured data point if that point agrees with 
the model prediction within the model prediction 
interval. For a model with negligible lack of fit, the 
95% prediction interval half width, PIHW 95 0/o , is 
approximately equal to 2 a where a is the standard error 
in the fit. For example, suppose a = 100 K and the 
model prediction at a given site is 1000 K. There is no 
more than a 5% probability that a confirmation point 
would fall outside the range of 800 K and 1200 K due 
to ordinary chance variations. However, just as we 
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Fig. 7. The minimum percentage of trials expected to be 
successfully predicted if there is at least a 99% probability 
that the number of successful confirmations is consistent with 
the 95% prediction interval of an adequate model. 

would not require 100 tosses of a coin to produce 
exactly 50 heads in order to declare the coin fair, 
neither do we require exactly 95% of all confirmation 
points to fall within the 95% prediction interval. 
(Surely 94 successes out of 100 trials would not 
invalidate the model, just as it is not unlikely that 96 
points might fall within the 95% prediction interval 
half- width in any one set of 100 confirmation trials.) 
Just as it happens that 99% of the time, a fair coin will 
produce between 37 and 63 heads in 100 trials, so it is 
that there is a 99% probability that in 100 trials, 89 or 
more confirmation points (not 95!) will fall within the 
95% prediction interval of an adequate model. For a 
lower number of trials, say 1 0 trials, only seven or more 
successes are required. Figure 7 reveals the minimum 
percentage of successful confirmations required of a 
model for there to be at least a 99% probability that this 
percentage is consistent with the 95% prediction 
interval of an adequate model. This percentage is 
graphed in Fig. 7 versus the number of trials. 

The payoff for doing all this extra work - 
designing the experiment, acquiring the data in random 
order, and then analyzing the data with RSM - can be 
profound. First, there is usually a substantial 
improvement in accuracy, particularly, avoidance of 
systematic error because of randomization. Second, the 
entire data set can be compactly represented by an 
equation and a handful of constants that can be used by 
the customer to compute the dependent variables 
anywhere in the parameter space. Third, and most 
important, there is a marked improvement in precision 
compared to conventional methods because of using 
response-surface methodology. 

To illustrate the improvement in precision possible 
with MDOE, Fig. 8 compares MDOE to a conventional 
experiment where replication has been used in both 
experiments to improve measurement precision. All of 
the curves assume the same number of data points 
(1000) and same standard deviation in the data (100 K). 


Note that for measurements with errors that are 
normally distributed about a mean of zero, the 95% 
confidence interval half-width of an n-point sample is 
approximately 2c/(n 1/2 ). The corresponding value for a 
fitted surface varies over the surface, but its average 
value scales as p 1/2 . See Eq. (1). For a small number of 
measurement locations (for example, three), the 
conventional single-point method provides comparable 
uncertainty to an MDOE response surface fit having 
three parameters. In this case, the 1000 points are 
spread equally among three sites and the 95% 
confidence interval half width in both cases is -10 K. 

Suppose, however, that the flow may have a 
substantial amount of spatial structure that one would 
like to map. If the number of measurement points is 
increased, the number of measurements per location 
decreases accordingly. Consequently, the uncertainty 
estimated from data acquired at only one location will 
have a corresponding increase in measurement 
uncertainty. On the other hand, the MDOE method 
maintains a constant measurement uncertainty, 
assuming that the same three-parameter model is used. 
If a higher order model is used to fit a more 
complicated flow structure, the simulation shows that 
MDOE still provides a marked increase in precision 
compared to the conventional single-point statistics 
method. For the case of a 10-parameter model used to 
fit data obtained over 1 00 independent spatial locations 
there is more than a factor of three improvement in 
measurement uncertainty. Alternately, MDOE could 
obtain the same measurement uncertainty as the 
conventional method with nearly 1/1 0th the number of 
data points! That amounts to a substantial cost savings. 

In this paper, we use the term “costs savings” 
loosely to mean reduction in the number of 



Fig. 8. Comparison of measurement uncertainty (95% 
confidence interval half widths) between MDOE and 
conventional single-point statistics as a function of the 
number of spatial locations probed. The total volume of data 
is held constant at 1000 points. A 100 K standard deviation is 
assumed. 
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measurement points required. The real costs of doing 
experiments can be computed many different ways, and 
often involve a large initial investment to build the 
facility, for example. A comprehensive analysis of all 
such costs is beyond the scope of this paper. 
Additionally, it is important to note that randomization 
and replication usually reduce the rate of acquisition of 
data, offsetting some of the reduction in costs. 
However, in many prior applications of MDOE, 
substantial cost savings have been achieved. 5 

One might question how simply fitting a curve 
through the data would increase its precision. It 
appears too good to be true. The key point is that by 
fitting a curve through the data, information is shared 
from neighboring spatial locations. Suppose you have 
obtained temperature measurements at five locations 
along a line where the temperature is slowly increasing. 
If another measurement is desired between two of the 
existing points, it would not take many measurements 
to establish whether the new location agrees with the 
trend established by the other points. Information is 
shared and fewer measurements are required to achieve 
the same uncertainty. Using the single point method, 
the experimentalist starts “from scratch” at each new 
spatial location and many measurements are required to 
reduce the uncertainty in the mean. 

Considerations for Mapping Temperature in a 
Supersonic Combustor 

The description above outlines the way an ideal 
experiment would be designed, executed, and analyzed 
using MDOE. Unfortunately, some compromises that 
prevented the full implementation of MDOE were 
required in the present experiment. In this section, we 
briefly describe the experimental setup as it relates to 
the current paper and then discuss how MDOE was 
implemented. 

Experiments were performed in NASA Langley 
Research Center’s Direct-Connect Supersonic 
Combustion Test Facility (DCSCTF), which is a 
vitiated, blowdown wind tunnel. The supersonic 
combustor model consisted of a short rectangular duct 
containing a rearward- facing step, after which gaseous 
hydrogen fuel was injected. The supersonic combustor 
operated with a steady flow time of 10-20 seconds. 

We used planar BoxCARS to make single- shot 
broadband nitrogen CARS temperature measurements 
in the combustor. 12 During the flow time, 100-200 
single-shot temperature measurements were obtained. 
CARS spectra, acquired from an intensified CCD linear 
array attached to a spectrometer, were fit with a library 
of theoretical curves for a range of temperatures and N 2 
concentrations to determine the gas temperature on 
each pulse of the laser. Thus, a database of x, y, z 
locations, temperatures, and N 2 concentrations was 


generated for five measurement planes in the flowfield 
for unpiloted operation and three planes when H 2 pilot 
fuel was injected upstream of the main fuel injection. 
A series of measurement in a stable Hencken flat-flame 
burner determined that the precision of the CARS 
measurement technique was ±6% of the measured 
temperature (standard deviation of 3%). The flame 
measurements reported in Ref. 12 showed that the 
CARS system produced temperatures that were 
systematically high compared to calculated values by 
~150 K in the rich region of a hydrogen/air flame. 
Good agreement was found for stoichiometric and lean 
flames. However, this experiment was recently 
repeated and this systematic error was not observed. 
The mean measured temperatures agreed with 
computed values to within 70 K on average over a 
range of stoichiometries from 0.5 to 4.0 and no 
systematic bias in the measured temperatures was 
observed. 13 The systematic bias observed in Ref. 12 is 
thought to be caused by spatial nonuniformities present 
in the flame, though this point is still under 
investigation. 

Figure 9 shows the resulting temperature maps 
obtained in the supersonic combustor. 12 Briefly 
summarizing the results, the vitiated air flow enters the 
test section at about 1160 K. For the unpiloted case, 
cold fuel, with a stagnation temperature of about 300 K, 
is injected between planes 1 and 3. Evidence of a small 
amount of combustion occurs around the periphery of 
the fuel jet in plane 3 where the temperature exceeds 
the incoming freestream value. The temperature of the 
cold jet increases between planes 3 and 5, but most of 
the combustion occurs between planes 5 and 6. Planes 
6 and 7 show hot combustion products on the top and 
bottom of the duct and remnants of the cold fuel jet 
near the middle of the duct. 

The current paper describes in greater detail how 
MDOE was applied to this experiment and how the 
analytic surfaces were fit to the data, to produce the 
temperature maps shown in Fig. 9. The analysis in this 
paper is limited to plane 3, unpiloted operation, which 
was thought to be the most challenging measurement 
plane to analyze due to the large temperature gradients 
present. 

The objectives for the present experiment were 
determined from interviews with the customer, a 
computational fluid dynamics expert who has 
subsequently computed the flow. 14 Together it was 
decided that the goal of the present experiment would 
be to map the flowfield with a specified precision, in as 
many planes in the combustor duct as resources would 
permit. From discussions with the CFD expert, we 
established ±50 K as the required 95% confidence 
interval for model predictions. For resource planning 
purposes, we computed the volume of data necessary to 
produce a model with such a 95% confidence interval, 
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Piloted 

Unpiloted 



Fig. 9. CARS temperature maps in the combustor duct. See 
Ref. 10 for details. 


averaged over all the points used to fit the model. Also, 
from prior measurements in the same facility 15 we knew 
that turbulent fluctuations in the flow had a standard 
deviation of about 300 K. Furthermore, observing the 
spatial distributions in past measurements, we 
determined that we would probably fit the data with 
response surface having approximately 20 parameters. 
Using Eq. (1) the required number of data points was 
estimated to be 2880 per measurement plane. 

If we had instead used the conventional approach 
of acquiring many temperature measurements at a 
single location to reduce uncertainty in the mean, and if 
we targeted the same precision requirements, 144 
temperature measurements would have been required at 
each site. The MDOE budget of 2880 points per plane 
then corresponds to only 20 measurement locations if 
the conventional method had been used. These would 
perhaps be enough spatial locations to measure the 
temperature along two or three lines in the flowfield. 
But, for the same cost, MDOE allows us to map the 
temperature over the entire plane. 

The data acquisition rate was 10 Hz and the tunnel 
typically operated for 10-20 seconds per run, so that 
-100-200 shots could be acquired per run. Thus a 
minimum of 20 runs would be required per 


measurement plane to meet the precision requirements. 
Because the tunnel must be allowed to cool down 
between runs, we were limited to four or five runs per 
hour. The CARS apparatus and the DCSCTF require 
about an hour each of set-up and takedown time. So, an 
entire day was just enough time to map the temperature 
in a single plane with the required measurement 
precision. We obtained between 2000 and 4000 
instantaneous temperature measurements per plane. 

Given a data rate of one plane per day, our budget 
limited the number of measurement planes we could 
acquire. Budget constraints included real dollar costs 
for facility time, cost for the H 2 gas, as well as 
availability of staff to run the tunnel, run the CARS 
system, and ensure the safety of the workers and 
facility. The stated objective then was “to map the 
mean temperature in five or more planes of the 
supersonic combustor with a measurement precision of 
better than ±50 K with 95% confidence.” In the end, 
seven measurement planes were acquired over 1 0 days 
of operation. This occurred over about two months of 
calendar time. 

The next aspect of the experiment design to be 
considered was site selection. Ideally, we would have 
selected sites according to the D-optimal design, 16 
which has several features that make it a desirable 
measurement pattern. Among other benefits, the 
29-point D-optimal design, shown in Fig. 10, has 
symmetry properties that maximize the accuracy of fit 
coefficients used in response surface methodology. 
Ideally, the CARS measurement volume would be 
translated to one of these locations and a single 
temperature measurement would be obtained. Then the 
measurement volume would be translated to one of the 
other sites, chosen at random, where a second 
temperature measurement would be obtained, and so 
on. Each location would be revisited many times, 
providing the number of replicates indicated at the 
locations in the figure. 

Unfortunately, in the present experiment the typical 
time required to translate between points in the 
measurement volume is about 20 times longer than the 
0.1 seconds between measurements. This data rate of 
10 Hz is set by the repetition rate of the laser. If we 
were to use the full randomization preferred by MDOE, 
our data rate would be cut by a factor of 20, which 
would have prevented us from achieving our precision 
goals without exceeding our resource budget. 

A compromise solution was found that 
implemented as much randomization as possible. Two 
types of measurements could be performed at the 
maximum data rate. One type of measurement was 
obtained by scanning the probe volume horizontally or 
vertically through the flowfield using a system of 
periscopes driven by stepper motors. A second type 
was performed at fixed locations in the duct for the 
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Fig. 10. D-optimal design for determining optimal locations 
for spatial measurements. 

entire duration of the run. The fixed location 
measurement scheme potentially could have been 
applied in a D-optimal site selection. However, 
performing the entire design would have required 29 
runs that would have produced far more data than 
required for the precision requirements. We believed 
that decreasing the number of set points by a factor of 
two, to match the precision requirements, would have 
sampled the flowfield too sparsely. So, a hybrid 
method was adopted where both types of measurements 
- fixed and scanned - were used. Thus, the D-optimal 
design was abandoned in favor of a rectangular grid. 
This was supplemented by six to eight fixed position 
measurements, depending on the measurement plane. 
All fixed locations were distributed uniformly in the 
vertical direction on the spanwise center of the flow. 
We alternated randomly between fixed and scanned 
runs. We also randomized the directions of the scanned 
measurements (up/down or across/back). Figure 11 
shows the sites selected for the different measurement 
planes in the experiment. For completeness, it should 
be mentioned that blocking was not invoked in the 
present experiment because no block effects were 
identified. 

After the experiment was performed, temperatures 
were computed from the raw CARS spectra. See Ref. 
12 for details of this procedure. The x, y locations of 
the measurement volume and the corresponding 
temperatures from all runs in a given plane were written 
to data files, one line for each individual temperature 
measurement. 

Two different statistical analysis programs were 
used to fit the data: Design-Expert® 16 and 


Piloted 

Unpiloted 



Fig. 11. Site selection for the experiment. 

TableCurve®3D. 17 These programs have many 
common features, including automatic fitting of 
surfaces to data and automatic computation of the 
statistical figures of merit (. Model F -statistic, R 2 , etc). 
TableCurve®, however, does not allow model terms to 
be removed as easily as Design-Expert®. Removing 
model terms is very important in the present application 
because reducing the number of terms p increases 
measurement precision, according to Eq. (1). It is often 
found that more than half of the terms in the models are 
insignificant. So, by removing these terms one can gain 
an increase in precision of a factor of about 1.4. 
Unfortunately, the only candidate model functions in 
Design-Expert® are polynomials, whereas TableCurve® 
provides thousands of different model forms, including 
Chebychev, Fourier, Sine and Cosine Series, etc. Thus, 
both software packages were used in the analysis. 

Nine different proposed models were evaluated 
prior to settling on the best one. For ease of 
explanation, these will be described in order of 
increasing complexity, rather than the order in which 
they were performed. Six different proposed models 
were fit to the entire data plane using Cartesian 
coordinates: 3 rd -, 4 th -, 5 th -, and 6 th -order polynomials 
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and two different versions of the Cosine Series 
Bivariate Order 6 Function. One proposed model 
partitioned the measurement plane into four subregions 
and used lower order polynomial models in each 
region. A transformation from rectangular to polar 
spatial coordinates was used for the other two proposed 
models, which were fit with 3 rd -order polynomials in r 
and 0. One of these polar coordinate analyses involved 
transforming the dependent variable. 

During the analysis of the data, three extreme 
residuals were removed out of a total of 2829 
measurements. There were perhaps 50 smaller 
residuals that were nonetheless candidate outliers, but 
these were retained because we could not say with high 
confidence that these were likely to be the result of 
spurious measurements. 


Results 


Polynomial Fits to Entire Cartesian Surface 

Equation (1) shows that the lower the order of the 
model (the smaller the p) the better. So the first 
proposed model was a 3 rd -order polynomial in the two 
independent variables, x and y: 


T = a + bx + cy + dx 2 + exy + fy 2 
+ gx 3 + hx 2 y + ixy 2 + jy 3 


( 2 ) 


This equation has 10 free parameters p. Note that it 
contains cross terms (e.g., xy, x 2 y, and xy 2 ) whose 
magnitude describes the interactions between the 
independent variables. Fitting Eq. (2) to the 
temperature measurements from plane 3 yielded 
unsatisfactory results. Four of the model terms were 
insignificant and were removed. While the overall 
model passed the F test indicating reasonable signal to 
noise, it did not pass the lack of fit tests. Visually 
comparing residuals showed several systematic trends, 
confirming a poor fit. Furthermore, the temperature 
surface looked unrealistic (concave up). 

The order of the fitting polynomial was increased 
sequentially to 4 th -, 5 th -, and 6 th -order. Each time, the 
model F test passed and the fit improved. These trends 
are illustrated in Fig. 12. Notice that the standard error 
in the fit, o, decreases with increasing order number: 
more terms allow the model greater flexibility to 
explain trends in the data. Also note the increase in the 
computed CIHW 95 o /o . This trend is caused by the 
increase in p as the model order increases. From (1) it 
is apparent that p 1/2 is growing faster than a is 
decreasing. Recall that the computed CIHW 95 o /o is only 
valid if the model fits the data (which none of these do). 
The 6 th -order model, which has 1 8 significant terms, is 
the only one of these models that passes its statistical 


lack of fit test. The 6 th -order polynomial temperature 
map is shown in Fig. 13(a). Upon close inspection of 
the model predictions and the residuals, important 
systematic discrepancies between the data and the fit 
were discovered. The worst problem occurred at the 
center of the cold fuel jet where the model predicted 
-350 K whereas the gas temperature measured with 
CARS was -250 K. Clearly there is some localized 
lack of fit. It is worth noting that we suspect lack of fit 
in the model even though it passed the statistical lack of 
fit tests. This is probably because there is 

comparatively little data in the fuel jet. 

The overwhelming majority of the measurement 
plane is fit satisfactorily, and a small (but important) 
part is not. This is an example of why possessing 
subject knowledge is very important for performing 
data analysis, and particularly for model fitting. 

For comparison with CFD, accurate modeling of 
the fuel jet is critical. For this reason, this 6 th -order 
polynomial fit was deemed unsatisfactory. The next 
logical step would have been to use a 7 th -order 
polynomial model. Unfortunately, the software did not 
allow higher than 6 th -order polynomial models. So, 
another method for improving the fit needed to be 
found. The ANOVA figures of merit for the 6 th -order 
polynomial and the rest of the fits discussed in the 
paper are summarized in Table 1. 

Polynomial Fits to Sub-Zones 

A method that is commonly used in fitting complex 
surfaces is to break the surface up into smaller sub- 
spaces. Each subspace can then be fit with a lower- 
order model, each having fewer parameters, potentially 
resulting in improved fits and reduced CIHW 95 o /o . We 
use this approach and partitioned the measurement 
planes into four sub-regions, or zones, as illustrated in 
Fig. 14. Zones 1 and 3 were adequately fit by 3 rd -order 
polynomials. Zone 4, which contains the fuel jet, 
required a 4 th -order polynomial fit, which did a very 



d Order 4th Order 5rth Order 6th Order 
Polynomial Polynomial Polynomial Polynomial 

Fig. 12. Trends in a, CIHW 95 % and LOF p-statistic with 
increasing polynomial order of model. 
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satisfactory job of fitting the fuel plume. Zone 2 
provided an interesting result: this region had very little 
curvature compared to the chance variations in the data. 
The overall model F test failed on all polynomial 
models attempted in this region. This indicates that 
there is no significant trend in the data with x or y, 
compared to the variability in the data. In this case, the 
entire sub-region is best modeled by the mean of the 
data in that region. 

In order to compare the quality of this fit to the 
others reported in the paper, an approximate CIHW95% 
was computed by weighting the CIHW95% values for the 
individual regions by the number of points in each 
region and dividing by the total number of points fitted. 
Table 1 shows that the method fails to reduce the 
CIHW95% compared to the 6 th -order polynomial. 
However, it is a better fit than the 6 th -order polynomial 
because it more accurately captures the most important 
features of the flowfield. Figure 13(b) shows the 
resulting temperature map. One drawback of 
partitioning into sub-zones is evident from the 
temperature map: discontinuities in temperature exist at 
the borders of the zones. This is unaesthetic, but not 
unscientific. The model actually predicts that a given 
sample would occur between this surface mean and the 
±PIHW95%. This is illustrated in Fig. 15 where a 
horizontal slice through the measurement plane is 
compared with the model. This slice crosses zones 1 , 4, 
and 3, in that order. The correct interpretation of this 
figure is that the model predicts with 95% confidence 
that seven out of 10 new measurements (as described 
by Fig. 7) would fall between the dashed lines in the 
figure. 


Model 

P 

Fstat 

PIHW 

CIHW 

R 2 

6 th order polynomial 

18 

230 

All 

37 

0.59 

Polynomial fits to 
four partitioned 
zones 

<14 

- 

- 

38 

- 

Cosine Series 
Bivariate Order 6, #1 

28 

185 

450 

45 

0.64 

Cosine Series 
Bivariate Order 6, #2 

13 

425 

447 

30 

0.64 

Polar coordinate 
transformation 

6 

898 

471 

22 

0.61 

Polar coordinate and 
ln(T) transformation 

6 

1038 

479 

22 

0.64 
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Table 1. Summary of fit results for select models. PIHW and 
CIHW are based on 95% confidence. All models represented 
in this table pass the LOF F-statistic and P-statistic tests, 
except where noted in the text. 


100 

x [mm] 

Fig. 13. Temperature maps resulting from fitting different 
proposed models, (a) 6 th -order polynomial, (b) polynomial fits 
to four sub-regions, (c) 27 term cosine series bivariate order 6 
function, (d) 12 term cosine series bivariate order 6 function, 
(e) polar coordinate transformed data fit with 3 rd order 
polynomial. 
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Fig. 14. Partitioning of measurement plane into four sub- 
zones. 



60 80 100 120 140 
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Fig. 15. Comparison between model prediction (four-zone 
polynomial model) and single-shot temperatures for spanwise 
cut through the flow at y = 76.8 mm. 


Cosine Series Bivariate Model 

In an attempt to model the temperature distribution 
with a more suitable basis function than polynomials, 
the TableCurve® program was used to find a suitable 
model. A promising model was the Cosine Series 
Bivariate Order 6 function. This function was chosen 
because it offered a compromise of a very good fit to 
the data (based on Model F-statistic , lack of fit F- and 
P-statistics , R 2 , and standard error in the fit, a) while 
having fewer parameters than many of the other 
functions. This function is similar to the first few terms 
of a Fourier series but contains only cosine functions 
and has products of cosines as well as the usual cos(x), 
cos(2x), cos(3x), etc., terms. The full 6 th -order model 
has 28 parameters. The fit produced by TableCurve® is 
shown in Fig. 13(c). 

The ANOVA results for this model suggested that 
more than half of the model parameters were 
insignificant. Removing these terms would decrease 
the CIHW95%o substantially. So, a data file was created 
that would allow Design-Expert® to implement this fit. 
This was achieved by creating a spreadsheet that 
contained a column for each of the 27 terms in the 
model. The functional shape of each term (e.g., 
cos(3x)) was then computed in each column. When the 
spreadsheet was read into Design-Expert®, a model that 
was first-order in each of 27 different independent 
variables was fit to the data. As expected, after 
insignificant terms were removed, only 12 terms plus 
intercept remained, so that p = 13. The resulting 
temperature map is shown in Fig. 13(d). The fit is 
similar to Fig. 13(c). However, because p has been 
reduced from 28 to 13, the accuracy to which each of 
the coefficients in the model has been determined is 
substantially higher. With fewer model parameters, 
there is more data per parameter from which to 
determine the values of the parameters. As shown in 
Table 1, the resulting precision is markedly improved: 
from ±45 K to ±30 K. Figure 16 shows the residuals 
plotted against the spanwise direction, x. The residuals 


show no significant trends versus x except at the 
extremes, confirming that the fit over most of the 
surface is good. 

Figure 17 compares model predictions and 
prediction intervals to the experimental data. The 
model fits the general trends of the data very well. 
However, the model does not capture every subtlety of 
the data. Higher order terms would be required to fit 
sharp discontinuities in the data. Such higher order 
models would require more parameters and would yield 
a higher CIHW 95 o /o . 

Variable Transformation: Polar Coordinates 

In an effort to reduce the parameter count even 
further, we attempted to take advantage of the near- 
radial symmetry of the fuel jet. We transformed the x, y 
coordinates into radial coordinates: r, 0, where r 2 = (x- 
x 0 ) 2 + (y-yof and 6= tan 1 ((y-y 0 )/(x-x 0 )) where care was 
taken to add 180 degrees to #when x < x 0 . Figure 18 
shows the result of the transformation. The center of 
the coordinate system was chosen to be at the center of 
the fuel jet (x 0 = 108.6 mm, y 0 = 76.1 mm). Such a 
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Fig. 16. Residuals plotted versus x. 
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Fig. 17. Comparison between model predictions and single 
shot temperatures for Cosine Series Bivariate Order 6 model 
#2. This is a spanwise cut through the flow at y= 76.8 mm. 
The thick solid line in the middle is the model prediction. 
The two thinner lines above and below the model prediction 
are the 95% confidence interval half widths. The thin lines 
furthest from the model prediction are the 95% prediction 
interval half widths. 

transformation greatly simplifies the functional form 
required to fit the data. For example, a temperature 
profile in the shape of an inverted cone would be very 
difficult to fit in Cartesian coordinates; it would require 
many high-order terms. But, in polar coordinates, an 
inverted cone maps to a planar surface that can be fit by 
the simple model, T = a + br. This variable 
transformation worked as advertised; the fit order could 
be reduced substantially - to 3 rd -order - and the number 
of parameters could be reduced to six. This resulted in 
a low CIHW950/0 as shown in Table 1. Unfortunately, 
there were several undesirable consequences of this 
method. First, the model temperature trends towards a 
large negative temperature at small r, which is 
unphysical. The polynomial should be forced to have 
zero slope at r= 0. Similarly, when the data is fit in 
polar coordinates, it would be desired to have a periodic 
boundary condition in 6. 

A natural logarithmic transformation in the 
dependent variable (temperature) was found to correct 
the first problem. Fitting ln(T) instead of T forces the 
model to be better behaved at small r. This 
transformation also causes a significant improvement in 
the fit, as evidenced by the increase in R 2 in Table 1. 
Furthermore, it compares much better with the data in 
the fuel jet. The resulting temperature map is shown in 
Fig. 13(e). Comparing this map with the others in Fig. 
13 reveals that an artificial symmetry has been imposed 
on the data by the polar coordinate transformation. 
Perhaps this is caused by not retaining high enough 
order terms. However, we regard with suspicion the 
visible circular patterns, particularly the corkscrew 
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Fig. 18. Rectangular to polar coordinate transformation. The 
red crosshair marks xq, yo- 


shaped variation in temperature. The residuals provide 
further evidence of an unsuccessful fit. The residuals 
plotted against #show no significant trends, but the 
residuals plotted against radius, shown in Fig. 19, 
display some weak but significant trends. Even though 
this model passes all the statistical tests and has the 
smallest CIHW 95 o /o . , we believe that this model does not 
adequately represent the data. 

Discussion 

The various methods of fitting the data have their 
relative merits. While the 6 th -order polynomial fit was 
easy to perform, it produced a temperature map that 
disagreed with the experimental data in a critical region 
of the flowfield: the cold fuel jet. Partitioning the 
surface into four smaller surfaces allowed lower order 
polynomials to accurately fit the experimental data, but 
this process was very time consuming and it produced 
the second worst CIHW 95 o /o values of all the fits shown 
in Table 1. The Cosine Series Bivariate Order 6 
function fit the surface well with a minimum of effort. 
However, the large number of model parameters caused 
the fit uncertainty to be the worst in Table 1. This 
problem was corrected by removing the insignificant 



Fig. 19. Residuals plotted versus radius, r. 
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terms in a subsequent analysis, producing a CIHW 95 o /o 
equal to ±30 K. Inspection of residuals shows no 
substantial trends. The polar coordinate transformation 
of the independent variables, combined with the natural 
logarithmic transformation of the dependent variable 
produced a result that had excellent statistical 
properties, but imposed an artificial symmetry on the 
flowfield. Also this model shows some small but 
substantial trends when residuals are plotted against 
radius. We conclude that the model that most 
accurately represents the measured temperatures is the 
cosine series bivariate function with insignificant terms 
removed. 

After passing these various criteria, the model is 
considered to be a “candidate model” for validation. 
The model was tested against a portion of the original 
data selected at random that had been set aside and not 
included in fitting the model (5%, or 140 points). 
According to Fig. 7, to validate the model, it must 
predict >90% of the trials correctly, or at least 126 out 
of the 140 trials. In fact, the model predicted 128 out of 
140 correctly (or 91%), so it was said to have passed 
the confirmation test. 

It is useful to compute whether the MDOE method 
did in fact reduce the volume of total data acquired in 
the experiment, as predicted by Fig. 8. In a similar 
experiment previously conducted in the same facility, 
Smith et al. 15 mapped the temperature in a different 
supersonic combustor. They determined the mean 
temperature in the conventional way by taking -70 
replicates at each of 63 points of a 9 by 7 grid. If they 
had had the same standard deviation (a = 223 K) as in 
the present experiment, they would have obtained 95% 
confidence level uncertainties in the mean values of 
temperature at each of those points of ±53 K. To 
further reduce their uncertainty to the ±30 K level 
obtained using MDOE would have required a total of 
225 measurements per location, or -14,000 points 
compared to the current experiment in which -2800 
points were obtained. MDOE allowed us to get the 
same uncertainty with a factor of five fewer points - a 
substantial reduction. 

In the current experiment, we could have reduced 
the data volume by a factor of (50/30) 2 or -2.5 and still 
met the precision requirements of ±50 K at 95% 
confidence. However, the opposite could also have 
been true: If the turbulence level had been twice what 
was anticipated prior to the experiment, we would have 
failed to meet our precision requirement. So the 
amount of data obtained in the current experiment was 
probably appropriate. It was ample to protect us against 
incorrect assumptions of the nature of the flow, yet it 
was not so great as to have incurred substantially more 
time and expense than needed to satisfy the quality 
objectives of the experiment. However, there is 
another, better, solution to the resource scaling 


problem. If the CARS spectra could be analyzed as 
they were being acquired, or in between tunnel runs, 
then a surface fit could be performed as the experiment 
progressed. The fit uncertainty would be constantly 
monitored. Once the precision requirements were met, 
the experiment could be terminated, having acquired 
the right amount of data. This approach would prevent 
acquiring too much data. The costs saved could then be 
used, for example, to study other measurement planes, 
fuel flowrates, model configurations, etc. 

In addition to the substantial cost savings, MDOE 
defended against systematic errors through 

randomization. In fact, we did see some trends in the 
residuals when plotted against time, a tell-tale sign that 
the kind of systematic variation against which 
randomization is intended to defend was in fact present 
during the experiment. During certain runs, for 
example, the gas temperature appeared to be 
systematically increasing with time. These trends need 
to be analyzed more closely. Nonetheless, randomizing 
the order that the data were obtained allowed us to 
defend against such bias errors. 

Another benefit of the response surface methods 
incorporated in MDOE is that the data can be 
compactly presented. This facilitates comparison 
between the data and theoretical or computational 
predictions of the flow, which is frequently an end use 
for data obtained using laser-based measurement 
techniques. For example, the surface fit for the 
13-parameter Cosine Series Bivariate Order 6 model is 
shown in Eq. (3): 

Temperature = 

1068.2 ± 

103.2 cos(Y') ± 

117.3 cos(2X’)± 

-91.9 cos(3Y’)± 

-180.8 cos(4X’) + 

36.5 cos(5X') (3) 

25.2 cos(5Y') ± 

-45.7 cos(X')cos(Y') ± 

-109.6 cos(2X’)cos(Y’)± 

143.2 cos(4X')cos(Y') ± 

191.2 cos^X^cospY') ± 

-48.2 cos(5X')cos(Y') ± 

139.5 cos(4X’)cos(2Y’). 

Unfortunately there are certain cultural barriers that 
are likely to retard the rate at which the laser-based 
measurements community adopts MDOE. The primary 
obstacle is the haste with which many scientists in this 
field tend to approach experiments. MDOE involves 
carefully planning an experiment, which can be time 
consuming. However, the time spent in designing a 
good experiment is negligible compared to the time 
spent trying to make sense out of a poorly planned one. 
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But designing an experiment with MDOE seems time 
consuming compared to the conventional method, in 
which the final design of the experiment often occurs 
moments before it is executed. Another barrier to 
usefully applying this method is that the user must learn 
about certain statistical methods (e.g., ANOVA) that 
might presently be unfamiliar. Still another barrier is 
the time required to analyze the data, which can 
frustrate the natural human tendency to demand a final 
result immediately upon completion of the experiment. 
Nonetheless, we believe that MDOE will gain 
increasing acceptance in the laser-based measurements 
community as more members of that community 
become proficient in its use and more familiar with the 
advantages of this method, which include lower costs, 
higher quality, and less time to achieve quantitative 
performance objectives. 

Conclusion 

We have applied modem design of experiments 
principles to a single-point laser based measurement 
apparatus for the first time at NASA Langley Research 
Center, and perhaps the first time outright. MDOE 
allowed us to scale the data volume to the customer’s 
precision requirements. One consequence of this 
scaling was a large cost savings of perhaps a factor of 
five compared to previous measurements performed in 
the same facility. Another consequence of using 
MDOE was the minimization of systematic errors. This 
was achieved by randomizing the order of data 
acquisition as much as possible. Using MDOE allowed 
us to map the mean temperature with a 95% confidence 
level of ±30 K. This is substantially lower than the goal 
of ±50 K stated prior to the experiments. In fact, the 
precision requirements could have been achieved had 
the data volume been reduced by more than a factor of 
two. 

We plan to more fully implement randomization in 
future experiments by redesigning the stepper-motor 
driven periscope system. Furthermore, we will 
investigate extending the analysis to fit CARS 
temperature maps in the third dimension (streamwise 
direction), creating an analytic model to predict the 
temperature in all three spatial coordinates within the 
duct. Another goal is to determine gas density from the 
intensity of the CARS signal. This is traditionally a 
difficult problem with CARS due to poor long-term 
optical stability, because drift in the system’s alignment 
can cause an apparent decrease in measured density. 
MDOE’s quality assurance tactics (randomization, 
replication, and blocking) are well suited to correct for 
this type of systematic bias error. 

With minimal modification, the method outlined in 
this paper should be applicable to many other similar 
experimental methods, such as Rayleigh and Raman 


scattering experiments, laser-induced thermal grating 
(LIT A) velocimetry, sound speed measurements, etc. It 
may even be possible to extend the method to imaging 
experiments, such as planar laser-induced fluorescence 
(PLIF), in an effort to improve measurement precision. 
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